-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding support for ARB (Application Resource Bundle) (.arb) format #338
Conversation
Tagging @kbairak for review, but feel free to tag others if needed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great job!
openformats/formats/json.py
Outdated
self.existing_keys.add(key) | ||
|
||
if isinstance(value, DumbJson): | ||
self._find_keys(value, key) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of going infinitely deep, we could instead search up to level two and collect entries that are non-text. This way we don't have to bother with escaping keys etc. Here is an idea; I haven't tested it so there might be a few errors:
class ArbHandler(JsonHandler):
...
def parse(self, content, **kwargs):
...
keys = self._find_keys(parsed)
stringset = self._extract(parsed, keys)
def _find_keys(self, parsed):
if parsed.type != dict:
raise ParseError("...")
keys, keys_to_ignore = set(), set()
for key, key_position, value, _ in parsed:
if not key.startswith("@"):
if not isinstance(value, str):
continue
if key in keys:
raise ParseError(
f"Duplicate string key ('{key}') in line "
f"{self.transcriber.line_number}"
)
keys.add(key)
else:
if isinstance(value, DumbJson):
(inner_value, _), = value.find_children('type')
if inner_value != "text":
keys_to_ignore.add(key)
return keys - keys_to_ignore
def _extract(self, parsed, keys):
stringset = []
for key, _, value, value_position in parsed:
if key not in keys:
continue
...
return stringset
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did not try this. Anyway, ARBs normally have only two levels of json, so we don't expect infinite nesting.
openformats/formats/json.py
Outdated
else: | ||
raise ParseError("Invalid JSON") | ||
|
||
def compile(self, template, stringset, **kwargs): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, the only reason you re-define compile
and _copy_until_and_remove_section
is so that you can add the keep_sections
functionality, which is only needed for testing. I am not very fond of this idea. It makes the code unnecessarily complex. However, there is a case where not removing the sections makes sense.
This is not adequately described in the docs, but when Transifex invokes this handler, it will pass is_source
as a keyword argument. In this case, we can be certain that
- the stringset will not be missing anything and that
- the user wants to get the exact same file they previously uploaded
So, if this is indeed the same use-case, maybe you can rename keep_sections
to is_source
.
In any case, lets make sure that when is_source=True
is passed to compile
, the end result is the same as the originally uploaded file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tried this, replaced keep_sections
with is_source
, but that did not work, unit-tests failed. Anyway, re-definition of the compile()
method is necessary now, as it handles language code for ARB.
openformats/formats/json.py
Outdated
raise ParseError("Invalid JSON") | ||
|
||
def _extract(self, parsed): | ||
if parsed.type == dict: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need for this check, we already did it in _find_keys
. Perhaps to make it clearer, lets make one check in parse
and then in _find_keys
and _extract
assume that parsed
represents a dict.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
There is another issue I am concerned about. Looking at the examples of the official Flutter documentation, it looks like the language files are supposed to be more bare: only key-translation pairs without any metadata. So, the question is: should we follow this? should we make sure that when Have you looked into this? |
73356ed
to
e8ad0fe
Compare
...
Did not implement this logic in the latest commit. It looks unusual if compared to other openformats handlers, which have their compiled files as similar as possible to the source files (only strings from stringset change). Maybe we could try to get back to this idea in a separate pull-request. |
continue | ||
|
||
context_key = f"@{key}.context" | ||
context_value = self.metadata[context_key] \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implemented issue 1 from QA spreadsheet
) | ||
new_template = self._clean_empties(new_template) | ||
|
||
if language_info is not None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implemented issue 2 from QA spreadsheet
context_value = self.metadata[context_key] \ | ||
if context_key in self.metadata.keys() else "" | ||
description_key = f"@{key}.description" | ||
description_value = self.metadata[description_key] \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implemented issue 3 from QA spreadsheet
Problem and/or solution
Adding parsing and compiling for ARB (Application Resource Bundle) (.arb) format
How to test
1. Running unit-tests
/openformats/tests/formats/arb/test_arb.py
contains tests for arbUse
pytest /openformats/tests/formats/arb/test_arb.py
to run tests2. Through testbed
Use "ARB" handler in the testbed
Reviewer checklist
Code:
PR: